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ABSTRACT 

A  preliminary  version  of  a  system  evaluation  questionnaire  was  trialed  concurrent 
with  a  mission  management  system  (MMS)  concept  demonstrator  test  during  June 
2000.  The  questionnaire  was  designed  to  examine  usability  and  interface  issues  relating 
to  the  design  of  the  systems  or  tools,  and  to  provide  suggestions  for  future  iterations  of 
the  system(s).  The  returned  questionnaires  were  examined  for  both  the  system-related 
information,  and  the  content  and  style  of  the  questions  themselves.  Although  some 
useful  information  was  gathered  using  this  questionnaire,  the  author  concludes  that 
such  questionnaire  methods  should  be  combined  with  (for  example)  behavioural 
observations  in  order  to  provide  a  more  complete  picture  of  optimal  system  design  and 
user  training  for  the  human-machine  context. 
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Human  Machine  Interface  and  Usability  Issues: 
Exploring  a  Preliminary  Mission  Management 
System  Evaluation  Methodology 


Executive  Summary 

This  report  outlines  an  examination  of  a  preliminary  evaluation  methodology  for  the 
human  factors  issues  involved  in  the  human-machine  interface  (HMI)  and  usability 
areas.  It  is  intended  that  this  be  the  starting  point  for  further  development  of  an 
evaluation  methodology  that  encompasses  not  only  HMI  and  usability  issues,  but  also 
the  wider  impact  of  introduced  technology  on  teams  themselves.  The  issues  addressed 
here  would  include  shared  situation  awareness  (SA,  for  both  inter-  and  intra-  team 
settings),  the  performance  and  workload  of  teams  and  their  members,  and  the  training 
needs  accompanying  the  implementation  of  automated  tools. 

The  questionnaire  was  based  around  issues  outlined  in  the  usability  and  HMI 
literature;  both  Nielsen's  usability  design  heuristics,  and  items  for  evaluating  cockpit 
automation  design  [outlined  by  Craig  and  Burnett]  were  adapted  for  use  in  this 
questionnaire.  Items  with  a  specific  military  focus  were  also  included,  as  the  evaluation 
method  is  initially  intended  for  application  in  the  military  context.  Questions  designed 
to  elicit  work  domain  expert  knowledge  were  also  included,  so  that  the  needs  of  the 
end  user  (in  this  case,  the  military)  could  also  be  examined,  and  suggestions  be  put 
forward  for  future  iterations  of  systems  being  evaluated. 

Conclusions  drawn  from  this  exploratory  work  include  the  following:  that  additional 
system-specific  detail  be  included  in  the  questionnaire  prior  to  use  in  subject 
evaluations;  that  a  larger  number  of  subjects  from  all  of  the  necessary  groups  (ie. 
technical  experts,  work  domain  experts,  and  novice  system  operators)  should  be 
consulted;  that  results  from  system  evaluations  should  be  used  to  determine  where 
more  intense  training  may  be  needed,  in  addition  to  highlighting  areas  which  may 
require  more  intensive  system  development  to  facilitate  user  productivity,  efficiency 
and  effectiveness;  and  that  questionnaire  information  is  not  adequate  in  and  of  itself. 
The  information  gathered  in  this  way  should  be  supplemented  by  behavioural 
observations,  including  timing  the  durations  of  specific  activities  to  ensure  that  a 
function  rated  'fast  and  easy  to  use'  is  not  simply  an  inaccurate  subjective  perception. 

In  addition,  it  is  noted  that  evaluations  of  usability  should  -  in  this  context  -  be 
conducted  concurrently  with  evaluations  of  situation  awareness  (both  individual  and 
shared),  inter-  and  intra-  team  interactions,  and  workload  in  order  to  produce  a  more 
complete  picture  of  the  impact  of  automated  tools  on  the  operations  they  are  designed 
to  enhance. 
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1.  Introduction 


Investigating  the  usability  and  human-machine  interface  aspects  of  tools /systems 
designed  to  enhance  human  functioning  is  vital  to  the  augmentation  of  human- 
machine  partnerships  in  the  military  context.  International  Standard  134071  clearly 
defines  usability  in  the  following  terms: 

"the  extent  to  which  a  product  can  be  used  by  the  specified  users  to  achieve 
specified  goals  with  effectiveness,  efficiency  and  satisfaction  in  a  specified  context 
of  use." 

This  is  further  clarified  by  the  following  definitions: 

Effectiveness:  "accuracy  and  completeness  with  which  users  achieve  specified 
goals". 

Efficiency:  "resources  expended  in  relation  to  the  accuracy  and  completeness  with 
which  users  achieve  goals". 

It  follows  that  evaluations  of  either  prototype  or  actively  employed  decision  support 
systems  (DSSs)  and  automated  situation  awareness  (SA)  tools  should  focus  on  these 
critical  issues.  It  is  beneficial  for  the  organisation  implementing  the  tools/systems  to 
ensure  that  these  are  not  only  operating  effectively,  but  that  they  do  not  add 
substantially  to  the  cost  or  difficulty  of  staff  training  and  skill  maintenance. 
Additionally,  the  users/staff  must  be  able  to  maintain  effective  and  efficient 
functioning  with  these  systems  under  the  potentially  extreme  conditions  within  which 
they  could  be  expected  to  operate^. 

Accordingly,  this  report  outlines  an  examination  of  a  'first-cut'  usability  and  human- 
machine  interface  (HMI)  evaluation  questionnaire.  The  questionnaire  is  designed  to 
address  a  series  of  issues  previously  identified  in  the  usability  and  HMI  literature  as 
important  for: 

♦  Facilitating  electronic  data  management  and  presentation; 

♦  Facilitating  the  efficient  interaction  of  humans  with  PC-based  systems; 

♦  Minimising  the  detrimental  effects  of  tool/ system  use  for  users. 

In  addition,  it  is  supplemented  with  items  designed  to  elicit  from  subjects  (ie.  users, 
observers,  and  work  domain  subject  matter  experts  [SMEs])  information  relating  to 
desirable  characteristics  for  future  system  designs. 


'  Human-centred  design  processes  for  interactive  systems  :  1999. 

2  In  the  case  of  military  users,  this  may  mean  a  large  variety,  including  (but  not  limited  to) 
desert  situations,  tropical  conditions,  and  snow /blizzard  conditions,  all  of  which  can  adversely 
affect  hardware  (and  therefore  software)  system  operations. 
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2.  Evaluating  tools  and  support  systems 

The  evaluation  of  such  tools  and  user  interfaces  from  the  human  factors  perspective  - 
where  the  results  of  evaluations  comprise  important  input  into  the  design  of  future 
iterations  of  tools /systems  -  forms  an  essential  component  of  the  user-centred  style  of 
design.  It  is  essential  in  this  process  of  user-centred  system  design  that  both  task 
requirements  and  context  of  use  be  considered.  It  follows,  then,  that  consulting  with 
the  users  also  forms  a  vital  part  of  any  evaluation  methodology.  When  evaluating 
systems,  the  needs  of  the  end  users  must  form  part  of  the  evaluation  criteria  to  ensure 
that  the  final  product  will  meet  these  needs. 

The  effective  evaluation  of  a  system  in  the  early  stages  of  implementation  is  illustrated 
in  the  work  reported  by  Huf,  Kieboom,  Hobbs,  Medlow,  Jaensch,  Barrowcliff  and 
Harvey  (2000),  where  a  usability  evaluation  was  conducted  on  the  Command  Data 
Network  System  (CDNS).  A  workshop  was  conducted  to  evaluate  this  system,  with  the 
participants  consisting  of  a  balance  of  usability  design  consultants,  technical  domain 
experts,  work  domain  experts  (ie.  Army  personnel),  and  participants  of  varying  levels 
of  experience  with  the  system.  The  evaluation  approach  taken  was  based  on  Nielsen's 
(1993)  user  interface  design  principles  (outlined  in  Table  1,  page  3),  and  was  designed 
to  assess  the  system  against  the  principles  of  interface  design  to  elucidate  any  potential 
problems  in  system  usability.  A  series  of  recommendations  emerged  which  were 
reported  to  the  system  designers  as  areas  needing  further  development. 


For  the  current  work,  a  list  of  'essential'  criteria  for  effective  systems  was  compiled 
from  the  general  usability  and  interface  literature  (including  Nielsen,  1993),  as  well  as 
from  the  lists  of  specific  display  information  and  the  general  question  areas  outlined  by 
Craig  and  Burnett  (1999).  Although  Craig  and  Burnett's  items  were  designed  for  and 
applied  in  the  context  of  cockpit  automation,  many  of  the  questions  and  issues  are  as 
applicable  to  a  ground-based  system  as  an  air-based  system.  Thus,  a  relatively  large 
number  of  questions  were  placed  into  the  following  categories  outlined  by  these 
authors:  display  readability,  screen  layout,  functionality,  general  information,  information 
processing,  spatial  awareness,  information  degradation,  fatigue,  as  well  as  an  overall  user 
rating  for  general  performance  on  the  system(s).  The  details  of  each  category  were 
tailored  to  include  issues  likely  to  be  of  interest  to  military  users,  as  initial  system 
evaluations  will  be  conducted  on  applications  designed  for  use  in  the  defence  arena. 
Thus,  questions  ranged  from  simple  display  issues  ("Information  is  organised  in  a  clear 
and  uncluttered  way"  and  "Colour  coding  makes  map  features  easily  distinguishable") 
to  the  maintenance  of  spatial  awareness  ("There  are  effective  cues  to  maintain  accurate 
spatial  awareness")  and  fatigue  issues  ("Information  overload  is  a  problem  for  the 
user").  Questions  with  a  specific  military  inclination  included  such  issues  as  the 
presence  or  absence  of  military  grid  references  on  displayed  maps,  and  a  library  of  the 
appropriate  symbology  for  military  use. 
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The  systems  currently  being  evaluated  using  this  preliminary  methodology  were  not 
initially  designed  with  this  list  of  categories  as  a  high  priority  {pers  comm.  Seymour,  R. 
S:  2001).  The  development  of  a  workable  functionality  within  a  given  time  and  cost 
frame  was  of  the  highest  priority,  and  thus  these  categories  were  given  less  emphasis 
than  they  would  have  been  given  in  a  production  system.  The  results  of  the  evaluation 
should  therefore  be  of  interest  in  terms  of  the  perceived  usability  of  systems  designed 
without  explicit  usability  specification  input  from  the  end  users.  It  is  not  always 
possible  for  usability  practitioners  to  have  direct  input  into  the  design  of  systems 
during  the  initial  developmental  stages.  Thus,  an  evaluation  methodology  that  can 
assess  the  functionalities  already  provided  and  suggest  improvements  (if  necessary)  to 
a  product  in  the  iterative  stages  is  a  valuable  tool  in  such  situations. 

Some  of  the  issues  outlined  above  were  also  referred  to  in  an  article  by  Kraiss  (1989), 
who  discussed  the  improvement  of  user  interfaces  and  the  design  of  effective  DSSs.  He 
states  that  it  is  vital  for  any  DSS  to  be  integrable  with  the  user's  own  cognitive 
processes,  as  the  aim  of  the  introduction  of  such  tools  is  the  extension  of  the  user's 
cognitive  decision-making  abilities  (Kraiss,  1989).  The  basic  DSS  configuration  consists 
of  a  user  with  direct  access  to  the  system  being  controlled,  who  can  exchange 
information  with  the  DSS  via  the  user  interface.  This  relationship  is  shown  below  in 
Figure  1.  As  this  figure  shows,  an  effective  interface  is  vital  to  ensure  that  information 
input  and  display  actively  facilitate  user  functioning. 

Table  1.  Nielsen's  User  Interface  Design  Principles  (adapted  from  Nielsen,  1993). 


Principle 

Comment 

Simple  and  natural  dialogue 
(ease  of  navigation) 

Ensuring  that  the  application  supports  the  user  navigating  through  the 
user  interface  in  a  way  that  is  obvious  and  conducive  to  getting  the  job 
done. 

Speak  the  user's  language 

Ensuring  that  the  terminology  is  based  on  users'  language,  not  system 
oriented  terms. 

Minimise  user  memory  load 

Ensuring  the  visibility  of  objects,  actions,  and  options. 

Consistency 

Ensuring  consistency  within  an  application  and  with  other  applications 
used. 

Feedback 

(indication  of  user  operation) 

Ensuring  the  application  gives  clear,  helpful  feedback  on  what  the 
system  is  doing  and  what  the  user  needs  to  do. 

Clearly  marked  exits 

Ensuring  the  user  can  exit  out  of  as  many  situations  as  possible. 

Shortcuts 

Ensuring  experienced  users  are  able  to  perform  frequently  used 
operations  especially  fast  using  shortcuts. 

Good  error  messages 

Ensuring  error  messages  are  clear,  concise,  and  constructive;  helping 
other  users  to  quickly  solve  the  problem. 

Prevent  errors 

Ensuring  that  the  application  reduces  the  opportunity  for  making  errors 
and  allows  users  to  readily  fix  errors  that  are  made. 

Help  and  documentation 

Ensuring  that  the  application  contains  usable  and  useful  support  tools, 
including  online  help  and  documentation. 
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Figure  1.  Basic  decision  support  system  cojtfiguration  (adapted  from  Kraiss,  1989). 

As  the  aim  of  the  introduction  of  such  technology  is  the  enhancement  of  user 
functioning,  questions  relating  to  the  support  of  information  management  and  decision 
making,  problem  solving  and  planning,  and  error  sensitivity  should  be  considered.  As 
there  was  limited  capacity  to  investigate  these  issues  in  the  context  of  this  preliminary 
study,  questions  regarding  learnability,  comparative  ratings  with  other  systems, 
difficulties  in  learning  the  system,  areas  which  may  be  lacking  in  terms  of  the  end 
user's  needs,  ease  of  system  awareness  maintenance,  and  suggestions  for  future  design 
improvements  were  also  included  in  order  to  expand  on  the  information  elicited  by  the 
questionnaire.  The  issues  described  above  must  be  addressed  in  order  to  create  a 
complete  understanding  of  the  HMI  issues  associated  with  the  introduction  of  new 
technologies,  and  the  functionalities  which  may  yet  be  lacking  in  the  tools  provided. 
These  issues  will  be  addressed  in  future  work  using  a  subject  pool  that  includes 
domain  experts,  a  revised  set  of  questions,  and  additional  observational  techniques^. 

Essentially,  then,  what  is  needed  for  the  effective  application  of  automation  to  the 
military  (and  indeed,  any)  field  of  operations  is  a  user-centred  approach  which  takes 
the  work  domain  into  account  as  an  important  part  of  the  "cognitive  system"  (Flach, 
Vicente,  Tanabe,  Monta  &  Rasmussen,  1998).  In  this  context,  the  needs  of  the  user  are 
examined  and  the  introduced  technology  is  the  instrument  with  which  these  needs  can 
be  met  and  human  functioning  enhanced.  That  is,  this  approach  seeks  to  make  the 
interaction  between  the  humans  and  the  work  they  do  optimally  efficient  and  effective. 
Thus,  there  must  be  meaningful  evaluation  of  the  interface,  the  tools'  functioning  (in 
terms  of  end  user  needs),  and  the  effect  the  tool  has  on  the  user's  performance  (beyond 
the  first  introductory  stages).  These  need  to  be  carried  out  from  a  human  factors 
perspective,  and  not  simply  from  the  viewpoint  of  outright  performance  scores.  This 
will  enable  distinctions  to  be  made  between  purely  system-based  functioning  flaws 
and  the  difficulties  caused  by  ineffective  human-computer  interactions. 


These  observational  techniques  are  currently  under  development  and  will  be  included  in 
publications  following  experiments  conducted  in  March  2001. 
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Difficulties  may  arise  in  using  evaluations  comprised  of  the  subjective  opinions 
gleaned  from  questionnaires.  Attempting  to  implement  potentially  contrary 
suggestions  of  different  users  and  domain  experts  could  conceivably  result  in 
conflicting  characteristics  listed  as  necessities  for  the  systems/tools  under 
consideration.  These  issues  should  be  dealt  with  in  terms  of  a  specific  'user  needs' 
focus  by  addressing  the  questions  of  which  combinations  of  characteristics  appear  to 
enhance  user  functioning,  and  which  combinations  produce  optimal  performance 
under  a  wide  range  of  conditions.  Here  it  is  evident  that  initial  evaluations  (such  as  the 
method  outlined  in  this  report)  will  lead  to  further  investigations  of  usability  and 
interface  issues,  and  changes  in  future  system  designs. 

In  keeping  with  the  team-centred  structure  of  military  operations,  future  investigations 
in  this  vein  should  also  include  the  impact  of  the  graphical  user  interface  (GUI)  on 
groups  of  users  rather  than  dealing  solely  with  individuals.  That  is,  work  on 
'groupware'  for  multi-user  systems  should  form  at  least  one  focus  of  future  work 
applied  in  the  military  context.  Researchers  have  begun  studying  aspects  of  such  multi¬ 
user  systems  (see  Calvary,  Coutaz,  &  Nigay,  1997),  however  this  appears  to  primarily 
be  from  the  perspective  of  the  imderlying  system  architectures.  Work  has  expanded 
into  the  area  of  visualised  coordination  support  for  teams  involved  in  distributed 
decision  making  (eg.  Svenmarck,  1998),  where  systems  implementations  are  intended 
to  provide  more  efficient  ways  of  coordinating  individuals  in  complex  dynamic 
systems  (pp.  103;  see  also  Brehmer,  1988).  Combining  this  evaluation  methodology 
with  assessments  of  team  performance  should  -  in  future  -  provide  a  more  in  depth 
understanding  of  the  complexity  of  technology  introduction  into  a  team  situation. 

3.  Aims 


In  essence,  the  aim  of  this  study  was  to  assess  the  utility  of  a  questionnaire 
methodology  for  evaluating  the  user  interfaces  and  general  usability  of  automated 
planning  and  decision  support  tools.  Additionally,  any  useful  information  elicited  from 
the  users  and  observers  will  be  applied  to  future  iterations  of  the  evaluation 
methodology,  and  as  suggestions  for  developers  of  the  systems/ tools. 

4.  Methodology 


4.1  Subjects 

Due  to  the  limited  availability  of  subjects,  two  objective  observers  (one  a  military  SME, 
and  one  with  a  relatively  low  level  of  experience  in  the  military  arena)  and  system 
experts  on  two  of  the  individual  tools  were  able  to  complete  the  questionnaires  and 
answer  the  questions. 
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4.2  Apparatus  and  tools 

A  mission  management  system  (MMS)  concept  demonstrator  -  involving  five  different 
automated  information  representation  and  handling  tools'*  -  was  examined  using  both 
objective  observers'  and  subjective  system  users'comments  and  opinions.  Information 
regarding  the  users'  level  of  experience  with  the  systems  was  also  obtained  as  part  of 
the  questionnaire. 

The  MMS  concept  demonstrator  is  a  working  combination  of  the  following  systems 
and  tools;  Battlescape,  Carmen,  Mapinfo,  BCSS  and  IME.  It  is  being  developed  under 
the  Land  Air  Systems  task  as  part  of  a  concept  demonstrator  Land /Air  system  of 
systems  (Seymour,  Sands,  Grisogono,  Unewisse,  Vaughan  &  Baumgart  2001). 
Experience  gained  with  this  concept  demonstrator  will  assist  DSTO  advice  to  the 
AIR87  project  (the  acquisition  of  ARHs). 

The  questionnaire  itself  was  designed  based  on  the  important  HMI  and  usability  issues 
discovered  in  the  literature.  46  separate  statements  regarding  aspects  of  the  systems 
were  rated  on  a  Likert  rating  scale  (where  1  represented  strongly  agreeing  with  the 
statement,  and  7  represented  strongly  disagreeing  with  the  statement).  This  was 
followed  by  a  general  rating  of  ease  of  use  on  a  1  to  10  rating  scale  (with  1  being  very 
easy  and  10  being  very  difficult).  Seven  supplemental  questions  were  placed  at  the  end 
of  the  questionnaire  to  provide  the  subjects  with  an  opportunity  to  record  their  own 
ideas  and  views  (in  their  own  words).  These  questions  related  to  learnability  of  the 
system,  comparative  ease  of  use,  main  difficulties  in  learning  to  operate  the  system, 
areas  which  may  be  lacking  in  terms  of  the  end  user's  needs,  maintenance  of  system 
awareness  amidst  distraction,  and  suggestions  for  aspects  of  future  system  design. 

The  objective  observers  were  given  the  questionnaire  fifteen  minutes  prior  to  the 
beginning  of  system  testing^,  and  asked  to  familiarise  themselves  with  the  types  of 
issues  they  would  need  to  rate.  They  then  watched  the  operators  as  they  used  the 
systems,  and  rated  each  of  the  four  available  systems  concurrently  according  to  their 
perceptions  of  the  systems'  performance  in  each  of  the  question  categories.  The 
observers  were  allowed  to  query  the  system  operators  to  clarify  any  aspects  that  were 
unclear  to  them.  The  system  operators  ('users')  were  also  given  the  questionnaires 
fifteen  minutes  prior  to  system  testing  and  asked  to  rate  the  system  they  were 
operating  from  their  more  knowledgeable  viewpoint.  This  was  carried  out  in  order  to 
provide  a  more  experienced  'hands-on'  person's  view  of  the  tools. 


■*  These  included  (in  no  particular  order):  BCSS,  Battlescape,  Smartboard  linked  with  Mapinfo, 
Carmen  and  Mimio  with  whiteboard.  Unfortimately,  Carmen  3.1  (the  updated  version)  was  not 
available  in  time  for  this  evaluation  and  so  was  omitted  from  the  assessment.  Brief  descriptions 
of  these  and  other  MMS  component  systems  are  available  in  Appendix  C. 

^  This  was  the  point  at  which  the  wargame  commenced  and  the  systems  were  online  and  being 
used  to  access,  input  or  display  data. 
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The  questionnaire  is  included  in  Appendix 
the  following  list: 

Category  Number: 

1 
2 

3 

4 

5 

6 

7 

8 
9 

4.3  Procedure 

The  questionnaires  were  collected  at  the  end  of  a  two-day  period  to  give  the  users  an 
opportunity  to  fill  in  all  the  questions.  Following  this,  the  ratings,  comments,  and  the 
style  in  which  the  questions  were  answered  were  examined  in  order  to  elucidate: 

♦  Observer  opinions  of  system/ tool  performance; 

♦  User  opinions  of  system/ tool  performance; 

♦  The  suitability  of  the  question  wording  for  eliciting  the  target  information; 

♦  Any  need  to  modify  the  length  of  the  evaluation  questionnaire; 

♦  What  types  of  information  are  still  missing/require  additional  elicitation. 

Consideration  was  also  given  to  other  potential  methods  of  tool/system  evaluation, 
and  work  to  be  conducted  in  the  near  future. 

5.  Results 


A.  Questions  were  categorised  according  to 

Category  Name: 

Display  readability 
Screen  layout 
Functionality 
General  information 
Information  processing 
Spatial  awareness 
Information  degradation 
Fatigue 

Overall  rating  of  system  ease  of  use 


5.1  Significance  of  results 

The  term  'significance'  is  used  here  to  indicate  a  meaningful  result  or  relationship 
between  two  variables.  The  alpha  level  at  which  significance  is  assumed  is  0.05,  which 
is  the  conventionally  accepted  alpha  or  confidence  level.  This  figure  indicates  the 
acceptable  probability  of  the  outcome  being  due  to  a  chance  result.  That  is,  when  alpha 
=  0.05,  there  is  an  acceptable  level  of  "chance"  of  5%,  indicating  a  probability  (p)  of  95% 
that  the  result  is  meaningful.  Thus,  if  a  statistical  test  returns  a  p  figure  below  the 
specified  alpha  level,  it  can  be  said  to  be  a  significant  result  at  this  level  of  confidence. 

For  example,  if  a  test  returns  a  result  of  (a)  p  =  0.03  (b)  p  <  0.04,  then  the  results  can  be 
called  significant.  The  lower  the  p  value,  the  more  significant  the  result.  A  p  value  of 
0.000,  for  example,  indicates  (by  the  three  decimal  place  convention  of  SPSS  that  the 


^  Statistical  Package  for  the  Social  Sciences. 


7 


DSTO-TR-1155 


probability  of  the  result  being  due  to  chance  is  less  than  0.00005.  This,  it  can  be  agreed, 
indicates  a  meaningful  result.  A  significance  (p-level)  of  (for  example)  0.380,  by 
contrast,  indicates  a  38%  probability  of  the  results  being  due  to  chance,  which  is 
unacceptably  high.  The  result  in  this  case  is  termed  non-significant. 

These  "significance"  conventions  will  be  used  throughout  the  results  section  below. 

5.2  Observer  and  user  ratings  of  the  systems^ 

It  was  interesting  to  note  that  the  ratings  given  to  systems  by  the  users  and  the 
observers  were  relatively  consistent  in  many  areas.  The  observer  and  system  user 
ratings  appear  quite  similar  for  the  display  readability,  screen  layout,  functionality,  general 
information  and  information  processing  categories  for  system  1.  This  is  shown  in  Figure  2. 
For  example,  general  preferences  in  the  way  information  is  displayed  included  large 
screen  displays:  subjects'  opinions  consistently  indicated  that  'bigger  is  better'.  Ratings 
data  from  the  questionnaires  are  shown  in  Appendix  B. 

Based  on  the  similarities  shown  in  this  graph,  the  Pearson's  r  correlation  coefficients 
were  calculated  separately  for  categories  that  displayed  similar  scores  for  all  raters 
together,  and  those  with  disparate  rater  scores.  Pearson's  correlation  coefficients  for 
system  1  ratings  on  question  categories  1  to  4  are  shown  in  Table  2  below. 

Table  2:  Pearson's  correlation  coefficients  for  systems  1,  question  categories  1  to  5. 


Rater 

Correlated  with  Rater 

Pearson's  r 
coefficient 

Significance 

Observer  1 

Observer  2 

0597 

0.000 

Observer  1 

User  1 

0.555 

0.000 

Observer  2 

User  1 

0.698 

0.000 
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^  Results  will  not  be  presented  in  terms  of  the  specific  systems  for  two  reasons:  (a)  this  is  simply 
a  preliminary  exploration  of  the  evaluation  methodology,  and  so  should  not  be  taken  as  a  final 
word,  and  (b)  the  BCSS  project  (and  thus  any  evaluation  result)  is  restricted. 
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Question  Category 


Figure  2:  Mean  ratings  for  system  1. 

Clearly,  there  is  similarity  between  the  ratings  given  to  the  system  by  the  observers  and 
the  system  user  on  categories  1  to  5,  with  a  highly  significant  positive  Pearson  r 
coefficient.  It  should  be  noted  that  these  categories  contain  question  types  concerning 
more  easily  observable  aspects  of  the  systems,  while  the  other  categories  rely  more  on 
the  users  experiences  with  the  system  in  order  to  make  a  judgement.  This  may  affect 
the  similarity  of  the  ratings,  as  the  observers  may  perceive  functions  which  apparently 
require  many  movements  or  a  larger  perceived  amount  of  time  to  perform  as  being 
more  difficult  to  use. 


It  can  be  noted,  though,  that  categories  6  to  9  {spatial  awareness,  information  degradation, 
fatigue  and  overall  system  ease  of  use  rating  respectively)  show  great  similarity  in  ratings 
only  between  the  observers  for  system  1.  The  correlations  between  observers  and  the 
system  user  are  non-significant.  These  coefficients  are  shown  below  in  Table  3.  The 
difference  in  user  and  observer  ratings  is  possibly  attributable  to  the  difference  in 
actual  experience  with  and  simply  observing  the  use  of  a  system  in  operation. 

Table  3:  Pearson's  correlation  coefficients  for  system  1,  question  categories  6  to  9. 


Rater 

Correlated  with 
Rater 

Pearson's  r 
coefficient 

Significance 

Observer  1 

Observer  2 

0.735 

0.016 

Observer  1 

User  1 

-0.613 

0.971 

Observer  2 

User  1 

0.225 

0.532 

The  ratings  shown  in  Figure  3  for  system  3  show  a  somewhat  different  pattern,  with 
similarities  between  all  raters  for  categories  1  to  4,  and  more  disparate  ratings  for 
categories  5  to  7.  Greater  similarity  between  all  raters  can  again  be  seen  for  question 
categories  8  and  9.  The  correlation  coefficients  are  shown  in  Tables  4,  5  and  6 
respectively. 
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Question  Category 


Figure  3:  Mean  ratings  for  system  3. 


Table  4:  Pearson's  correlation  coefficients  for  system  3,  question  categories  1  to  4. 


Rater 

Correlated  with 
Rater 

Pearson's  r 
coefficient 

Significance 

Observer  1 

Observer  2 

0.610 

0.000 

Observer  1 

User  2 

0.482 

0.004 

Observer  2 

User  2 

0.433 

0.011 

As  shown  in  the  tables  5  and  6  below,  categories  5  to  7  show  very  disparate  ratings, 
with  non-  significant  correlations  between  scores  on  these  categories.  Categories  8  and 
9  show  a  similar  pattern,  except  that  observer  2  and  the  system  user  do  tend  to  rate  the 
system  similarly  on  the  fatigue  and  overall  ease  of  use  scales  (as  displayed  by  the 
significant  positive  correlation  in  Table  6  below). 

Table  5:  Pearson's  r  correlation  coefficient  for  system  3  categories  5  to  7. 


Rater 

Correlated  with 
Rater 

Pearson's  r 
coefficient 

Significance 

Observer  1 

Observer  2 

0.344 

0.364 

Observer  1 

User  2 

-0.534 

0.139 

Observer  2 

User  2 

-0.451 

0.223 

Table  6:  Pearson's  r  correlation  coefficient  for  system  3  categories  8  and  9. 


Rater 

Correlated  with 
Rater 

Pearson's  r 
coefficient 

Significance 

Observer  1 

Observer  2 

0.677 

0.323 

Observer  1 

User  2 

0.812 

0.188 

Observer  2 

User  2 

0.978 

0.022 

It  can  be  expected  that  the  systems  scored  similarly  in  some  of  these  categories,  in  that 
users  of  computerised  systems  tend  to  have  similar  types  of  generalised  needs  and 
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preferences  with  respect  to  displays  and  layouts.  Easy  to  read  fonts,  clear  pictures, 
intuitive  operations,  and  lack  of  onscreen  clutter  are  -  for  example  -  common  desires  in 
most  users  of  automated  systems.  In  addition,  the  items  in  these  categories  (or  the 
direct  consequences  of  these  items)  can  easily  be  noted  by  observers,  so  there  is  little 
constraint  on  what  the  observer  can  see.  It  is  clear,  for  example,  when  the  symbols  and 
text  are  easily  read  and  the  colour  coding  is  clearly  understandable.  The  main 
constraint  on  this  observation  is  that  observers'  perceptions  of  ease  or  speed  of  use  can 
be  affected  by  operation  of  the  system  by  an  experienced /expert  user.  That  is,  a  bias 
towards  perceived  ease  can  form  on  watching  an  expert  at  work  if  the  observer  does 
not  take  this  into  accotmt. 

Discrepancies  in  subjects'  opinions  appeared  most  clearly  where  the  users'  experience 
with  the  systems  came  into  play.  The  categories  information  processing,  spatial  awareness 
and  information  degradation  resulted  in  quite  different  scores  for  the  two  systems  whose 
users  answered  the  questionnaire  compared  with  the  two  observers.  That  is,  the  users 
perceived  the  functions  in  these  two  categories  as  simpler  to  perform  than  did 
observers  with  little  or  no  experience  with  the  system.  This  difference  in  perception 
appears  to  be  a  direct  result  of  familiarity  with  the  system,  and  is  an  important 
indicator  of  areas  where  training  may  need  to  be  intensified.  This  difference  in  scores 
also  indicates  that  a  particular  grouping  of  subjects  for  evaluation  purposes  is 
important  in  order  to  gain  a  deep  understanding  of  the  system  and  the  training 
requirements  accompanying  system  implementation.  That  is,  experienced  users  may 
evaluate  a  set  of  system  functionalities  as  very  simple  to  use,  where  a  novice  may  find 
them  difficult  to  comprehend  and  operate.  This  type  of  information  is  highly  useful  in 
designing  training  protocols,  and  can  highlight  areas  in  which  tools  have  good 
functionality  but  in  which  it  may  take  longer  and/or  more  intensive  training  to  create 
user  proficiency. 

The  two  systems  rated  only  by  the  observers  showed  the  expected  differences  in  their 
patterns  of  ratings.  System  2  was  a  very  simple  and  basic  system,  which  both  observes 
rated  relatively  consistently.  System  4  was,  however,  more  complex.  Observer  1  had 
more  experience  with  the  system  and  communicated  more  closely  with  the  system  user 
while  taking  observations,  and  tended  to  rate  the  system  as  easier  to  use  than  did 
observer  2.  The  mean  ratings  for  each  system  by  the  two  observers  are  shown  in 
Figures  4  and  5  below. 
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Figure  4:  Mean  ratings  for  system  2. 


Figure  5:  Mean  ratings  for  system  4. 


Conducting  the  question  category  split  using  the  method  applied  previously,  the 
categories  showing  the  most  consistent  similarity  in  ratings  were  correlated,  as  were 
those  showing  consistent  dissimilarity  in  ratings.  This  produced  the  Pearson's 
correlation  coefficients  shown  in  Table  7  below. 

Table  7;  Pearson's  r  coefficients  for  ratings  on  system  2  between  observers  1  and  2. 


System 

Question 

Category 

Pearson's  r 
coefficient 

Significance 

2 

1-7 

0.225 

0.147 

2 

8&9 

-0.943 

0.057 

4 

1-4 

0.600 

0.000 

4 

5-9 

-0.202 

0.507 
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It  is  apparent  that  for  system  2,  the  ratings  given  to  question  categories  1  to  7  by  the 
two  observers  show  a  mild  positive  correlation  (which  is  nonetheless  non  significant). 
Thus,  there  is  a  similar  general  pattern  of  scores  on  the  given  items,  but  this  is  not  a 
particularly  strong  trend.  Ratings  for  categories  8  and  9,  however,  are  strongly 
opposite;  views  on  the  overall  ease  of  use  and  the  issue  of  user  fatigue  were  very 
different,  possibly  reflecting  the  individual  preferences  of  the  observers. 

Ratings  for  system  4  showed  a  similar  pattern  to  those  observed  in  systems  1  and  3, 
where  questions  categories  1  to  4  showed  strongly  similar  scores  with  highly 
significant  positive  correlations  (as  in  Table  7  above).  Again,  these  are  the  categories 
which  are  easily  observable  and  this  may  help  to  account  for  the  consistency  of  ratings 
between  the  observers.  As  stated  previously,  system  4  was  more  complex  than  system 
2  in  its  operations  and  the  methods  of  accessing  its  functionalities.  Observer  1  had  had 
more  experience  with  the  system  and  communicated  more  closely  with  the  system  user 
while  taking  observations  than  did  observer  2,  and  so  tended  to  rate  the  system  more 
positively.  The  differences  are  apparent  in  the  correlation  coefficients  generated  by 
comparing  scores  on  categories  5  to  9  for  the  observers:  a  Pearson's  r  of  -0.202  indicates 
the  different  directions  of  the  ratings,  although  this  pattern  was  not  strong  enough  to 
be  significant  (with  p  =  0.507). 

5.3  Question  content 

It  is  clear  that,  although  the  questionnaire  covered  many  of  the  more  basic  aspects  of 
the  systems'  functioning,  there  were  some  aspects  that  needed  further  clarification  and 
depth.  For  example,  the  first  question  -  regarding  the  organisation  of  information  in  a 
clear  and  uncluttered  way  -  elicited  ratings  with  caveats  from  two  of  the  subjects,  and 
a  straight  rating  from  two  subjects.  This  clearly  indicates  that  there  are  additional 
conditions  that  should  be  considered  as  part  of  this  question.  One  possibility  for 
amendment  includes  the  addition  of  several  'parts'  for  the  question.  For  example: 

Q  1.  Information  is  organised  in  a  clear  and  uncluttered  way. . . 

(a)  When  there  are  less  than  10  pieces  of  information  onscreen 

(b)  When  actively  updating  information 

(c)  When  displaying  multiple  categories  ofinformation 

Here,  a  variety  of  possible  alternatives  should  be  included  in  the  question,  so  that  a 
more  specific  judgement  can  be  made  about  the  capability  of  the  tool  in  question.  Thus, 
the  subjects  can  rate  the  tool  under  each  of  these  conditions,  and  clearly  isolate  tools 
from  one  another  in  terms  of  their  capabilities.  This  is  particularly  important  in  critical 
areas  such  as  information  display. 

There  is  also  clearly  a  need  for  further  space  to  be  included  in  the  questionnaire,  as 
some  questions  elicited  comments  in  addition  to  the  required  rating.  Space  was 
provided  at  the  end  of  the  questionnaire  for  notes  by  subjects,  however  this  would  be 
simplified  by  including  space  directly  below  or  adjacent  to  the  question  itself.  One 
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difficulty  is  providing  adequate  space  while  leaving  the  questionnaire  a  reasonable 
size. 

Questions  regarding  items  such  as  time  lags  in  update  rates  appear  to  need  more 
clarity  of  definition.  That  is,  some  definition  of  "time  lag"  should  be  included  in  the 
question  itself  to  enable  a  clear  judgement  by  the  subject(s). 

Questions  regarding  the  presence  and  effectiveness  of  editing  (or  other)  functions 
should  also  allow  for  the  subject(s)  to  outline  which  functions  are  present  (if  this  has 
not  been  outlined  previously  by  the  researcher),  and  which  of  these  are  effective  and 
easy  to  use.  Simplification  of  these  ratings  would  be  achieved  if  the  functions  were 
already  described  by  the  researcher  in  the  question  list:  however,  when  rating  multiple 
systems  simultaneously,  this  can  prove  to  be  a  cumbersome  manner  of  presenting 
information  to  the  subject(s). 

Generally  speaking,  then,  questions  should  be  more  closely  tailored  to  the  individual 
systems  being  evaluated,  encompassing  their  current  capabilities  while  leaving  room 
for  suggested  improvements.  This  can  be  difficult  when  there  are  multiple  systems 
being  evaluated  in  a  single  questionnaire,  and  so  a  compromise  should  be  reached  in 
terms  of  the  level  of  detail  included. 

5.4  Question  categories 

It  would  appear  that  additional  categories  would  be  useful  in  evaluating  DSSs  or  other 
automated  tools.  When  evaluating  planning  tools,  for  example,  it  would  also  be  useful 
to  gather  information  on  the  ease  of  error  correction  and  prevention,  the  ability  to 
override  computer  generated /edited  information  if  necessary,  and  the  ease  of 
accessing  'help'  information.  This  would  result  in  useful  information  regarding  the 
potential  efficiency  of  use  by  both  expert  and  novice  users,  as  well  as  the  backup 
structure  available  to  prompt  novice  users  when  they  experience  difficulties  using  the 
system. 

5.5  The  subject  pool 

A  subject  pool  with  a  distinct  composition  needs  to  be  accessed  in  order  to  carry  out  a 
full  evaluation  of  any  system,  particularly  those  with  specific  end  user  applications.  In 
terms  of  systems  or  tools  with  military  application,  work  domain  experts  must  be 
included  as  well  as  the  technical  domain  exerts  [those  experienced  with  the  system(s)] 
and  inexperienced  users  (in  order  to  gauge  the  potential  learning  curve  for  tool  use). 

The  systems  should  be  studied  from  all  aspects,  bearing  in  mind  the  eventual  aim  of 
enhancing  user  functioning  without  adding  to  the  cost  or  difficulty  of  user  training. 

5.6  General  comments  and  suggestions  for  system  design 

Some  general  suggestions  to  improve  the  usability  of  systems  intended  for  military  use 
can  be  extracted  from  the  questionnaires.  In  brief,  the  suggestions  include; 
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•  The  ability  to  edit  and  send  information  in  a  distributed  fashion  would  be 
an  advantage  (when  backbriefing  commanders,  for  example); 

•  The  ability  to  make  symbols  "on  the  fly"  would  be  very  useful; 

•  The  ability  to  generate  and  label  threat  domes  is  an  added  functionality  that 
is  considered  essential; 

•  An  "undo"  function  is  critical  as  a  time-saving  device; 

•  Clutter  should  be  reduced  as  much  as  possible  without  losing  the  vital 
pieces  of  information  onscreen:  this  may  mean  increased  screen  size  should 
be  considered  a  priority; 

•  Editing  functions  should  be  simple  and  intuitive; 

•  Templates  should  be  easily  accessible; 

•  Screen  resolution  must  be  sufficient  to  reduce  user  eye  fatigue; 

•  The  ability  to  draw  manually  on  a  Touchscreen'  type  display  and  have  the 
information  recorded  for  distribution  would  be  an  advantage. 

As  work  on  systems  with  these  functionalities  is  currently  in  progress,  future 
evaluations  will  need  to  integrate  this  information  and  expand  on  the  detail  of  the 
suggestions  in  order  to  differentiate  between  the  relative  quality  of  the  various  system 
options  (or  combinations  thereof). 

5.7  Future  research  and  evaluation  methodologies 

Research  is  currently  being  conducted  using  a  condensed  set  of  usability  questions  in 
addition  to  behavioural  observations  of  users  interacting  with  the  systems  in  question. 
Data  collection  will  also  include  the  time  and  the  number  of  movements  required  to 
correct  errors  and  perform  certain  actions  integral  to  the  intended  use  of  the  tools  being 
assessed,  as  well  as  trialing  a  behavioural  assessment  of  the  impact  of  the  tools  on  the 
situation  awareness  (SA,  both  individual  and  shared)  of  the  users. 

Future  uses  of  the  evaluation  methodology  will  also  need  to  focus  on  the  impact  of 
automated  tools  on  the  SA  of  teams  as  a  whole  (in  settings  where  groups  or  teams  of 
people  need  to  rely  on  the  system  or  tool  for  enhanced  information  processing 
abilities).  Enhancing  an  individual's  SA  does  not  automatically  enhance  the  SA  of  a 
team,  and  therefore  work  must  be  carried  out  to  ensure  that  team  or  shared  SA  is  not 
degraded  by  the  introduction  of  automated  tools.  Evaluation  tools  are  currently  being 
developed  for  this  very  purpose. 

6.  Conclusions 

This  report  outlines  a  preliminary  insight  into  the  usefulness  of  a  paper  system 
evaluation  methodology.  It  is  apparent  that  much  more  work  must  be  done  on  the 
details  included  in  the  evaluation  questionnaire,  and  the  suggestions  for  future  uses 
and  modifications  include  the  following: 
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♦  Additional  system-specific  detail  should  be  included  in  the  questionnaire  prior 
to  subject  evaluations. 

♦  A  larger  number  of  subjects  from  all  of  the  necessary  groups  (ie.  technical 
experts,  work  domain  experts,  and  novice  system  operators)  should  be 
consulted. 

♦  Results  from  system  evaluations  should  be  used  to  determine  where  more 
intense  training  may  be  needed,  in  addition  to  highlighting  areas  which  may 
require  more  intensive  system  development  to  facilitate  user  productivity, 
efficiency  and  effectiveness. 

♦  Questionnaire  information  is  not  adequate  in  and  of  itself.  The  information 
gathered  in  this  way  should  be  supplemented  by  behavioural  observations, 
including  timing  the  durations  of  specific  activities  to  ensure  that  a  function 
rated  'fast  and  easy  to  use'  is  not  simply  an  inaccurate  subjective  perception. 

♦  Evaluations  of  usability  should  -  in  this  context  -  be  conducted  concurrently 
with  evaluations  of  situation  awareness  (both  individual  and  shared),  both 
inter-  and  intra-  team  interactions,  and  workload  in  order  to  produce  a  more 
complete  picture  of  the  impact  of  automated  tools  on  the  operations  they  are 
designed  to  enhance. 

Evaluations  of  system/tool  learnability  can  be  examined  in  only  the  most  basic  way 
using  this  type  of  questionnaire  technique.  Such  evaluations  actually  require  their  own 
intensive  investigations,  which  will  then  feed  their  outcomes  into  the  development  of 
user  training  protocols  for  each  system. 


16 


DSTO-TR-n55 


7.  References 

Brehmer,  B.  (1988)  Organisation  for  decision  making  in  complex  systems.  In 
Goodstein,  L.  P.,  Andersen,  H.  B.,  &  Olsen,  S.  E.  (Eds)  Tasks,  errors,  and  mental 
models.  Taylor  &  Francis:  London. 

Baildon,  B.  and  Robertson,  S.  (1999)  Tailoring  ModSAF  for  constructive  simulation  of 
Australian  forces.  In  Advancing  Simulation  Technology  and  Training:  the 
Simulation  Technology  and  Training  (SimTecT99)  Conference,  29th  March  to  1st 
April,  1999,  Carlton  Crest  Hotel,  Melbourne,  Australia. 

Calvary,  G.,  Couta2,  J.  &  Nigay,  L.  (1997)  From  single-user  architectural  design  to 
PAC*:  A  generic  software  architecture  model  for  CSCW.  In  Human  Factors  in 
Computing  Systems:  Computer  Human  Interaction  Conference  Proceedings  22  -  217 
March,  1997:  Atlanta,  GA. 

Craig,  I.  R.,  &  Burnett,  G.  L.  (1999)  The  design  of  a  human  factors  questionnaire  for 
cockpit  assessment.  In  People  in  Control:  An  International  Conference  on  Human 
Interfaces  in  Control  Rooms,  Cockpits  and  Command  Centres.  University  of  Bath,  UK. 
Conference  Publication  number  463. 

Flach,  J.  M.,  Vicente,  K.  J.,  Tanabe,  Fumiya.,  Monta,  K.,  and  Rasmussen,  J.  (1998)  An 
ecological  approach  to  interface  design.  In  Proceedings  of  the  Human  Factors  and 
Ergonomics  Society  42'“^  Annual  Meeting  -  1998. 

Huf,  S.,  Kieboom,  H.,  Hobbs,  W.,  Medlow,  D.,  Jaensch,  V.,  Barrowcliff,  G.,  &  Harvey,  S. 
(2000)  Command  Data  Network  System  usability  evaluation.  DSTO-IP-0026. 

International  Standard  13407  (1999)  Human-centred  design  processes  for  interactive 
systems.  ISO  13407:1999(E). 

Kraiss,  K.  F.  (1989)  Human  factors  aspects  of  decision  support  systems.  AGARD 
Conference  Proceedings  No.  453:  Operational  Decision  Aids  for  Exploiting  or  Mitigating 
Electromagnetic  Propagation  Effects.  Paper  presented  at  the  Electromagnetic  Wave 
Propagation  Symposium,  San  Diego,  CA.  May  15  -  1  9, 1989. 


17 


DSTO-TR-1155 


Kirby,  B.,  Seymour,  R.  S.,  &  Flay,  R.  (2000)  Battlespace  visualisation.  In  The  Australian 
Battlespace  Digitisation  Symposium,  July  2000,  DSTO  Salisbury,  Australia. 

Nielsen,  J.  (1993)  Usabilihj  engineering.  Academic  Press:  Cambridge,  MA. 

Seymour,  R.  S.,  Grisogono,  A.  M.,  Unewisse,  M.,  Johnson,  W.,  Krieg,  J.,  &  Haub,  J. 
(2000)  An  overview  of  situation  awareness  research  in  Land  Operations  Division. 
Paper  presented  at  The  Australian  Battlespace  Digitisation  Symposium,  July 
2000,  DSTO  Salisbury. 

Seymour,  R.,  Sands,  D.  G.,  Grisogono,  A.  M.,  Unewisse,  M.,  Vaughan,  J.  &  Baumgart,  R. 
(2001)  Application  of  Network  Centric  Warfare  Concepts  to  a  Land-Air  System  - 
an  experimentation  approach.  In  The  6**^  International  Command  and  Control 
Research  and  Technology  Conference:  Collaboration  in  the  Information  Age. 

June  19  -  21, 2001:  Annapolis,  Maryland. 

Svenmarck,  P.  (1998)  Visualised  coordination  support  in  distributed  decision-making. 
In  Waern,  Y.  (Ed)  Cooperative  Process  Management:  Cognition  and  information 
technolog}/.  Taylor  &  Francis  Ltd:  London. 

Unewisse,  M.,  Gaertner,  P.S.,  Grisogono,  A.  M.,  &  Seymour,  R.  S.  (1999)  Land 
situational  awareness  for  2010.  In  Advancing  Simulation,  Technology  and 
Training:  the  Simulation,  Technology  and  Training  (SimTecT99)  Conference.  29 
march  -  1  April  1999,  Carlton  Crest  Hotel,  Melbourne,  Australia. 


18 


DSTO-TR-1155 


Appendix  A:  MMS  evaluation  questionnaire 

Note;  The  questionnaire  has  been  condensed  here  for  ease  of  presentation. 


Your  name: 


The  system  you  are  using:. 


Please  rate  your  level  of  agreement  with  the  following  statements  by  placing  a 
number  from  1  to  7  in  the  box  provided. 

The  scale  is  as  follows: 


1  =  strongly  disagree 

2  =  moderately  disagree 

3  =  mildly  disagree 

4  =  undecided 

5  =  mildly  agree 

6  =  moderately  agree 

7  =  strongly  agree 

If  a  statement  is  not  applicable,  please  place  "n/ a"  in  the  box. 


Display  Readability. 


1.  Information  is  organised  in  a  clear  and  uncluttered  way. 

2.  Symbols  are  easily  recognisable  (consider  both  size  and  colour). 

Comments: . 

3.  Any  on  screen  text  is  easily  read  (consider  both  colour  and  brightness). 

Comments: . 

4.  The  colour  coding  makes  map  features  easily  distinguishable. 

Screen  Layout. 


5.  Important  information  is  highlighted  /  stands  out. 

6.  Information  is  located  in  the  same  position  on  each  consecutive  screen. 

7.  The  map  scale  is  present  on  screen  and  correct. 

8.  The  necessary  zoom  levels  were  available  and  easily  controlled. 

9.  Any  menus  are  organised  in  a  logical  way  (eg.  alphabetically  or  by  function). 

10.  The  necessary  coordinates  are  present  on  screen  and  easily  legible. 

11.  Any  units  of  measurement  are  consistent  with  military  standards. 
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Functionality. 


12.  All  relevant/ necessary  information  is  contained  on  screen. 

13.  Necessary  information  can  be  accessed  quickly  and  easily  without  losing  the  user  in 

the  GUI. 

14.  There  is  a  clear  functional  relationship  between  the  windows,  menus  and  toolbars. 

15.  Windows  are  managed  consistently  throughout  the  system. 

16.  Manipulation  of  the  map  and  overlay  information  is  fast  and  simple  (compared 

with  other  graphics  editing  packages,  for  example). 

17.  Manipulation  of  the  maps  and  overlay  information  is  intuitive  (eg.  logical  and 

fluid,  similar  to  Windows,  etc). 

18.  Field  labels  and  icons  are  consistent  across  screens. 

19.  There  is  need  for  labelling  of  timelines  in  addition  to  the  pop-up  labelling  already 

present. 

20.  Information  can  easily  be  checked  before  being  entered. 

21.  There  is  a  need  for  an  effective  undo  /  redo  function. 

22.  There  are  precautions  in  place  to  prevent  mistakes. 

23.  There  are  clear,  concise  and  constructive  error  messages. 

24.  Warning  and  confirmation  dialogues  provide  adequate  feedback  to  users. 

25.  The  perishability  of  the  information  can  be  adequately  manipulated. 


General  Information. 


26.  There  is  a  time  lag  in  information/screen  updates. 

If  so,  please  estimate  the  average  lag  in  seconds 

27.  Symbols  are  easily  entered/ drawn.  Comment: . 

28.  There  is  a  need  for  a  log  of  events /history  of  information. 

29.  Administrative  messages  can  easily  be  filed. 

30.  Any  screen  dialogue  is  simple  and  to  the  point  (ie.  no  extraneous  rubbish). 

31.  Onscreen  dialogue  is  in  the  accepted  /standard  military  language. 

32.  There  is  need  for  a  library  of  friendly/enemy  symbols  to  be  readily  available. 

33.  There  are  shortcuts  available  for  experienced  users  to  speed  up  frequently 

performed  operations. 

34.  The  system  gives  adequate  feedback  on  what  the  user  is  doing/needs  to  do. 


YES  /  NO 


unfonnation  Processing. 


35.  Necessary  information  can  be  superimposed /removed  quickly  and  easily. 

36.  Information  can  be  easily  and  quickly  edited  and  sent. 

37.  Information  is  quickly  and  easily  downloadable. 

Spatial  Awareness. 

38.  There  are  effective  cues  to  maintain  accurate  spatial  awareness. 

39.  The  display  should  be  "north  facing". 
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Information  Degradation. 

40.  There  is  generally  too  much  redundant  information  present  while  operating 

the  system. 

41.  There  is  generally  too  much  obsolete  information  onscreen. 

42.  There  is  a  need  for  an  effective  edit  function. 

43.  The  user  can  easily  exit  from  a  situation  when  /  if  necessary. 

Fatigue. 

44.  Information  overload  is  a  problem  for  the  user. 

45.  The  performance  of  the  system  suffers  due  to  data  overload. 

46.  Extended  operation  of  the  system  by  a  single  user  would  result  in  significant 

levels  of  fatigue. 


OVERALL  rate  the  ease  of  use  of  the  system  on  a  scale  of  1  - 10 
(1  =  extremely  simple  to  10  =  extremely  diflicult/complex):  _ 


General  short  answer  questions: 

Name: .  System: . 

Experience  level  with  system(s): . 

1  .How  easy  was  the  system  to  learn  /  become  familiar  with?  Please  rate  any  of  the 
systems  you  are  familiar  with  on  a  scale  of  1  [very  easy]  to  7  [very  hard]. 

2.  How  does  it  compare  with  other  systems  in  your  experience? 

3.  What  -  if  any  -  were  the  main  difficulties  with  learning  to  use  this  system? 

4.  How  easy  would  it  be  -  in  your  opinion  -  for  a  person  with  possibly  limited 
experience  with  computer  systems  such  as  this  to  learn  to  use  it  efficiently? 

5.  Are  there  any  areas  that  you  think  /  know  are  lacking  in  terms  of  what  the  end  user 
(ie.  rrulitary)  will  want  from  the  system?  If  so,  what  are  they? 

6.  Is  it  easy  to  maintain  system  awareness  when  there  are  distractions/interruptions 
occurring?  eg.  excessive  ambient  noise,  interruptions  by  other  people,  etc. 

7.  Do  you  have  any  suggestions  for  future  design  of  this  system  in  light  of  its  intended 
use? 
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Appendix  B:  Tabulated  results  from  the 
questionnaires  administered. 
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33. 

- 

- 

- 

- 

2 

1 

1 

2 

5 

- 

34. 

7 

5 

3 

6 

1 

1 

1 

1 

»5 

3 

Information  Processing 

35. 

7 

6 

3 

7 

7 

6 

5 

5 

5 

6 

36. 

7 

6 

3 

7 

6 

4 

- 

- 

7 

5 

37. 

7 

6 

4 

7 

7 

6 

- 

- 

7 

5 

Spatial  Awareness 

38. 

7 

2 

6 

7 

7 

4 

6 

1 

- 

6 

39. 

7 

6 

7 

7 

7 

7 

7 

7 

7 

*3 

Information  Degradation 

40. 

1 

- 

6 

1 

2 

6 

6 

3 

- 

3 

41. 

1 

- 

2 

1 

2 

5 

6 

3 

- 

2 

42 

1 

5 

7 

3 

1 

2 

6 

1 

- 

>(•4 

43. 

7 

6 

2 

6 

7 

Na 

6 

6 

- 

5 

Fatigue 

44. 

1 

2 

5 

1 

4 

4 

5 

5 

- 

5/6 

45. 

1 

2 

5 

6 

- 

4 

- 

6 

5 

- 

46. 

- 

1 

7 

- 

6 

5 

6 

7 

7 

7 

Overall  rating  or  ease  of  use 

3*-' 

5 

8 

4 

3 

3 

6/7 

7/8 

- 

8 

*1:  Objects  can  be  labelled  easily  but  don't  have  pop-up  labels. 

*2;  Already  has  a  library  of  symbols. 

*3;  It  is  north  facing. 

Has  an  effective  edit  function,  rated  6/7. 

*3;  Visual  feedback. 

Screen  gets  cluttered. 

*3;  An  experienced  operator  would  have  no  problem  adjusting  to  this  system. 
*3;  Once  developed,  need  not  be  redone. 

No  graphics  yet,  but  simple  bitmaps  available,  so  is  good. 

*1°:  Has  system  logs. 

Has  undo  function. 

*^3;  Has  undo/redo. 
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Appendix  C:  Descriptions  of  system  components 


Landscape. 

The  Mission  Planning  sub-system  of  the  MMS  concept  demonstrator  relies  on  user 
interaction  with  Landscape,  a  system  developed  inhouse  based  on  Autometrics  Edge 
Development  Option  that  was  3D  visualisation  for  the  digital  map  display.  Landscape 
gives  a  realistic,  scalable,  3D  representation  of  the  earth's  surface  using  a  variety  of  CIS 
data  including  digital  elevation  models,  raster  data  such  as  satellite  imagery  and 
scanned  maps  and  vector  data  for  representing  linear  features  such  as  hydrology, 
transport  networks,  contours  and  boundaries.  In  addition.  Landscape  can  display  text 
annotations,  map  and  military  symbology  as  well  as  customised  3D  models  and 
graphics.  For  example,  weapons  systems  can  be  graphically  represented  and  their 
effective  ranges  displayed  in  3D. 

LSAP  (Land  Situation  Awareness  Picture). 

LSAP  has  been  produced  under  contract  by  Maptek  and  is  a  3-D  visualisation  toolkit 
used  to  produce  3-D  visualisation  applications,  similar  to  Landscape.  It  is  a  multi¬ 
process  system  where  extra  applications  are  designed  to  provide  complex  output  by 
putting  layers  on  top  of  the  Carmen  core  and  multiple  executables  provide  the  overall 
application. 

The  Situational  Awareness  Picture  (SAP)  is  the  most  important  feature.  It  is 
constructed  from  a  terrain  and  an  optional  draped  image  (a  geolocated  image  with 
intersects  which  can  be  draped  over  the  terrain)  and  all  the  entities,  and  is  the  vehicle 
thru  which  operators  visualise  plans  and  entities.  Line  of  sight  tools  can  be  applied 
here.  The  planning  for  a  helicopter  pilot  can  therefore  combine  terrain  features  and 
entity  information,  and  predict  from  this  an  optimal  path  in  the  SAP.  This  can  then  be 
published  to  the  helicopter  itself  as  a  flight  plan  (Kirby,  Seymour  &  Flay,  2000). 

Mapinfo 

Mapinfo  is  a  COTS  product  which  is  a  comprehensive  desktop  mapping  tool  that 
enables  users  to  perform  complex  geographic  analyses  such  as  redistricting,  linking  to 
remote  data,  dragging  and  dropping  map  objects  into  applications,  creating  thematic 
maps  that  emphasise  patterns  in  the  data,  for  example.  It  enables  live  ODBC  access  to 
remote  databases,  cartographic  legends,  Seagate  Crystal  reports  (industry  standard 
report  writing  program),  and  grid  surface  thermatic  mapping.  It  is  compatible  with  MS 
Office  products,  and  lines,  symbols  etc  can  be  added  for  drawing  overlays. 

The  Smartboard  is  a  large  screen  onto  which  maps  and  overlays  shown  in  Mapinfo  or 
Battlescape  can  be  projected  for  clear  and  simple  viewing  by  users.  The  large  screen 
enables  an  increased  number  of  people  to  view  the  screen  simultaneously  (cf.  normal 
computer  screen  viewing).  It  also  employs  a  touchscreen  facility  to  allow  users  to  draw 
at  will  on  the  displayed  map,  although  this  cannot  be  saved  and  shared  with  other 
distributed  users  at  this  point  in  time. 
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Mimio  with  whiteboard 

This  is  an  electronic  whiteboard  attachment  that  allows  the  user  to  draw  plans, 
symbols,  overlays  etc  on  a  whiteboard  (using  the  specifically  designed  pens)  and  have 
these  automatically  appear  on  a  computer  screen.  These  overlays  can  then  be  saved 
and  passed  on  to  other  staff  for  use  with  their  own  maps.  It  utilises  technology  to  allow 
users  to  draw  in  large  format  on  a  whiteboard  and  then  have  the  product  transferred 
quickly  and  easily  to  an  electronic  format  which  can  be  saved  and  sent  to  multiple 
recipients. 

BCSS 

The  Battlefield  Command  Support  System  is  a  command  support  system  which  is 
currently  being  trialed  and  developed  by  the  Australian  Army.  The  Ops  component  of 
the  BCSS  system  enables  users  to  send  and  receive  text  messages,  and  to  view 
electronic  maps  which  display  the  entities  on  the  battlefield  and  update  the  positions  of 
these  during  the  battle. 

IME 

This  is  an  Information  Management  Engine  with  an  Oracle  database.  It  stores  the 
information  necessary  to  produce  the  entities  displayed  on  systems  such  as  CARMEN 
and  Battlescape. 
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